If you’ve used or considered using the Mozscape API to retrieve link metrics data, we’ve got something unique to share – a brand new beta of a much faster, more robust API. This beta version currently has just a few testers (and we’re seeking more), but thus far, we’re seeing remarkable results.
Carin, who manages the big data team here at Moz, helped share the story with me last week:
- The current API is not able to support everyone’s use case! Some people need to make a lot of calls in a really short period of time – our API currently can’t support more than 10 requests/second (even for paid users). Others have a large list of URLs they want to update metrics on every new index release – our current API doesn’t support batching very well and will timeout with batch sizes larger than 50 URLs.
- The beta version has made some serious performance improvements with single URL throughput and can handle 200 requests / second – the beta API is seeing a 400% throughput improvement, although response times will still be the same
-
To address batching users we’ve developed a new batching model – online batching (available in the beta API) and offline batching (coming soon to the beta API)
- Online batching: the maximum amount of results we can process in a POST without a timeout from S3. This has been improved from 50 URLs to 500 URLs in one batch request
- Offline batching (still in development): for batch sizes larger than 500 URLs, offline batching will process through the entire list (probably up to a certain limit not yet decided) and return a downloadable CSV link to S3 where all the data will be available. Since this is still in development, it is not clear the SLA on offline batching, but this feature will be also be available for beta testing as soon as it is feature complete!
Mozscape’s API is pretty big today – we served 154,352,249 (154 million) requests in the first 10 days of August and returned 1,186,736,774 (1.2 billion) rows of link metrics data.
You can still sign up free or try our paid API, but if you have serious demand for high-volume or large batches of link data, we’d love to have you in the beta for the new API. Just contact Andrew Dumont – [email protected] – and he’ll get you set up!
